The test of generic systems (and especially software applications) is a very important activity in their development process. The test is basically aimed at verifying whether each software application behaves as expected; this is of the utmost importance for validating the correctness, completeness and quality of the software application.
The test generally includes a series of test cases (for one or more test scenarios relating to specific operations of the software application). The execution of each test case involves the use of a predefined input for the software application; the software application generates a corresponding output in response to this input. A result of the test case is determined by comparing the actual output provided by the software application with an expected output thereof being deemed correct (with the test case that is set to “passed” when the two values match and it set to “failed” otherwise).
A problem of the tests is the identification of the expected outputs of the different test cases. Sometimes, the expected outputs may be set directly by a tester according to the logic of the software application. In more complex situations, instead, it is not possible to establish the expected outputs a priori (such as when they depend on unknown environmental conditions); a typical example is that of an asset management application that must discover the hardware configuration of a large group of machines. Therefore, the expected output of each test case is generally assumed to be equal to the actual output of a previous execution of the same test case; in these situations as well, however, the expected outputs must be somehow validated by the tester.
Anyway, the tester needs to participate in the definition of the test cases. This is a very critical task, which must be performed substantially in a manual way by the tester. Particularly, the direct identification of each expected output requires the tester to enter its value (for example, by means of a text editor); on the other hand, the identification of each expected output based on the previous execution of the corresponding test case requires the tester to verify whether this value is actually correct. Therefore, the design of any test involves a heavy human intervention; moreover a deep investigation activity is necessary (especially when the tester must validate the expected outputs based on the previous executions of the test cases).
The problem is particular acute in large software applications, all the more when they must be tested with a high level of accuracy. Moreover, the same drawbacks are magnified in very dynamic environments; indeed, in this case the same operations must be repeated at any new version, release or service level of the software applications.
In any case, the work required to design the tests is very time consuming. Moreover, the obtained results are not repeatable and prone to errors (being strongly dependent on the skill of the tester).
All of the above adversely affects the cost and the quality of the tests; this hinders their extensive application, with a negative repercussion on the reliability of the software applications.